chore(mcp): fix stale steps in the MCP tool authoring docs by webjunkie · Pull Request #111784 · PostHog/posthog

webjunkie · 2026-10-05T10:21:15Z

Problem

Agents that follow the MCP tool authoring docs write YAML that fails validation, or skip a step that CI then fails on.
The docs still show mcp_version, which the strict YAML validator rejects and nothing reads.
They also leave out the tool schema snapshot refresh and the UI apps generation step.

Stacked on the codegen fail-loudly PR. Part of a devex program on MCP codegen.

Changes

implementing-mcp-tools skill: removes mcp_version and the "Two MCP versions" section. Adds the snapshot refresh and generate:ui-apps steps after hogli build:openapi. Fixes the TRAILING_ACTIONS link.
services/mcp/README.md: the single tool is exec, not posthog. "Adding New Tools" points to the skill and handbook guide first.
services/mcp/src/tools/README.md: now covers hand-written tools only. Drops the feature flag example that referenced files and APIs that do not exist, and fixes the feature id format (snake_case) and the test path.

Docs only. No restructuring on purpose.

How did you test this code?

Docs change, so no automated tests. Ran hogli lint:skills and markdownlint on both READMEs locally.
Checked the new steps against services/mcp/CONTRIBUTING.md, scripts/yaml-config-schema.ts and .github/workflows/ci-mcp.yml.
The snapshot step is covered by the unit tests in CI. The CI workflow has no separate snapshot step. The UI apps step has its own check.

Test rationale: No test needed. The change is documentation and the linters above cover its format.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Release status

No feature flag controls this change

Docs update

This PR is the docs update.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code. Code written by Claude Sonnet 5.5 (claude-sonnet-5-5) subagents, orchestrated by Claude Opus 5.5 (claude-opus-5-5).

CodeRabbit CLI pass: no findings.
Skills invoked: /reviewing-with-coderabbit, /writing-pr-descriptions. /writing-skills was read for the skill edit.

posthog · 2026-10-05T10:21:21Z

FLASH MODE - Faster, but stupid, use regular ReviewHog for a heavy review

🦔 PostHog Review reviewed this pull request

Found 0 must fix, 1 should fix, 0 consider.

Published 1 finding (view the review).

github-actions · 2026-10-05T10:23:26Z

🤖 CI report

⚠️

Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

posthog · 2026-10-05T10:25:33Z

FLASH MODE - Faster, but stupid, use regular ReviewHog for a heavy review
PostHog Review alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

posthog

FLASH MODE - Faster, but stupid, use regular ReviewHog for a heavy review

PostHog Review

Found 1 should fix.

webjunkie · 2026-10-05T10:33:51Z

@codex review

chatgpt-codex-connector · 2026-10-05T10:34:08Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-05T10:40:48.172436Z	`fc8b4c9`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

coderabbitai · 2026-10-05T10:38:40Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used

📚 Code guidelines (5)

.agents/security.md — configured
.agents/skills/implementing-mcp-tools/SKILL.md — configured
.agents/skills/implementing-mcp-ui-apps/SKILL.md — configured
docs/published/handbook/engineering/ai/implementing-mcp-tools.md — configured
.agents/skills/writing-code-comments/SKILL.md — configured

📝 Walkthrough

Walkthrough

The MCP documentation now describes the CLI wrapper as exec, distinguishes generated tools from hand-written tools, and updates the hand-written tool quick start and definition guidance. The implementation skill adds a snapshot refresh step, limits UI-app regeneration guidance to YAML definitions that use ui_apps, corrects a link, and removes MCP version-specific instructions.

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to 32c59

The hand-written guide omits a check that helps keep new action tools’ compact names correctly rendered. This is a bounded documentation risk rather than a current runtime failure, so the PR is low risk with that follow-up.

Architecture Summary

Architecture risk: 🔵 Low · up to 32c59

The change affects 1 system.

Changed systems: services/mcp

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

observed — services/mcp (service) was modified; 2 changed files map to changed impact.

Before / after behavior

observed — Modified behavior in services/mcp/README.md: The cli-mode schema guidance is now attributed to the exec tool instead of the posthog tool.
observed — Modified behavior in services/mcp/README.md: The cli-mode server description now names the single wrapper tool exec instead of posthog.
observed — Modified behavior in services/mcp/README.md: The mode table now identifies exec as the single tool wrapping all tools in cli mode; the tools mode description is unchanged.
observed — Modified behavior in services/mcp/README.md: The tool-development instructions now direct readers to the generated products/<product>/mcp/tools.yaml workflow and its skill and handbook guides, and refer to src/tools/README.md only for hand-written tools. This replaces the general README-only guidance.

🚥 Pre-merge checks | ✅ 1

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description explains the problem, user-visible documentation changes, test rationale, release status, docs update, and agent context. It is complete enough to review; the missing session link and …

✨ Finishing Touches

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

services/mcp/src/tools/README.md-33-33 (1)
33-33: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Verify that feature_flags is registered.

The previous review found that the feature filter uses flags, not feature_flags. If that remains true, clients requesting ?features=flags will not discover tools copied from this example. Check the current filter and catalog; use flags here or register and test feature_flags.
#!/bin/bash
set -euo pipefail

rg -n -C 4 'feature_flags|"flags"|normalize.*feature|feature.*normalize' \
  services/mcp/src/tools/toolDefinitions.ts \
  services/mcp/schema/tool-definitions.json \
  services/mcp/README.md \
  services/mcp/tests/unit/tool-filtering.test.ts

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
Review profile: QUIET
Plan: Enterprise
Run ID: 93bec23c-07b8-4d1c-95f9-091540b3bc54

📥 Commits

Reviewing files that changed from the base of the PR and between b9ff2d3 and fc8b4c9.

📒 Files selected for processing (3)

.agents/skills/implementing-mcp-tools/SKILL.md
services/mcp/README.md
services/mcp/src/tools/README.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 5 remain after this review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc8b4c97c7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

posthog

FLASH MODE - Faster, but stupid, use regular ReviewHog for a heavy review

PostHog Review

Found 1 consider.

trunk-io · 2026-10-05T10:53:28Z

_{View Full Report ↗︎ ⋅ Docs}

Agents follow these docs literally. Several steps led to wrong output or a red CI run. - `mcp_version` is rejected by the strict YAML validator and nothing reads it, so the key and the "Two MCP versions" section are gone. - The authoring flow did not mention the tool schema snapshot refresh or `generate:ui-apps`. CI fails on stale output for both. - The `TRAILING_ACTIONS` link did not resolve from the skill folder. - The README named the single tool `posthog`. It is `exec`. - `src/tools/README.md` described a feature flag example that does not exist, used hyphenated feature ids, and pointed at a moved test path. It now says it covers hand-written tools only and links to the YAML flow for the rest.

Review follow-ups. The hand-written tool example used `feature_flags`, but the feature filter id for flags is `flags`, so a copied tool would be dropped for `?features=flags`. The authoring steps also missed that a tool behind a new `feature_flag` needs the flag in the snapshot test map, or it gets no snapshot and `generate:ui-apps` fails.

The hand-written tools guide sent authors to the repository README, but the MCP feature list lives in services/mcp/README.md.

posthog

FLASH MODE - Faster, but stupid, use regular ReviewHog for a heavy review

PostHog Review

Found 1 should fix.

coderabbitai

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

services/mcp/src/tools/README.md-51-51 (1)

51-51: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Document the action-tool naming check.

Line 51 covers kebab-case only. When an action tool can render a domain ending in an unknown operation verb, the author must add that verb to TRAILING_ACTIONS and test it in services/mcp/tests/unit/instructions.test.ts. Add this check to the authoring steps.

As per coding guidelines, “Whenever you add or rename an action tool, check the TRAILING_ACTIONS set” and cover it in services/mcp/tests/unit/instructions.test.ts.

Source: Coding guidelines

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
Review profile: QUIET
Plan: Enterprise
Run ID: 8e68ad52-de49-428d-8c57-c4fe4a61491e

📥 Commits

Reviewing files that changed from the base of the PR and between fc8b4c9 and 32c5974.

📒 Files selected for processing (2)

.agents/skills/implementing-mcp-tools/SKILL.md
services/mcp/src/tools/README.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 0 remain after this review.

stamphog

Approved.

Docs-only change to MCP authoring guides, with no runtime, schema, or CI behavior touched. The remaining bot comments are minor documentation nits.

coderabbitai[bot], posthog[bot] reviewed the current head.

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	0L, 0F substantive, 295L/3F incl. docs/generated/snapshots — within ceiling
tier	✓	T0 auto-approve: T0-deterministic (295L, 3F, two-areas, chore)
stamphog 2.3.1		`.stamphog/policy.yml` @ `32c5974` · reviewed head `32c5974`

trunk-io · 2026-10-05T14:50:26Z

This pull request was merged into master as part of stacked PR 111811.

deployment-status-posthog · 2026-10-05T15:24:44Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-10-05 15:24 UTC	Run
prod-us	✅ Deployed	2026-10-05 15:38 UTC	Run
prod-eu	✅ Deployed	2026-10-05 15:40 UTC	Run

webjunkie self-assigned this Oct 5, 2026

webjunkie added this pull request to stack #111785 October 5, 2026 10:21

posthog Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread services/mcp/src/tools/README.md Resolved

webjunkie marked this pull request as ready for review October 5, 2026 10:32

Copilot AI balanced review requested due to automatic review settings October 5, 2026 10:32

Copilot started reviewing on behalf of webjunkie October 5, 2026 10:32

View session

Copilot AI reviewed Oct 5, 2026

coderabbitai Bot reviewed Oct 5, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread services/mcp/src/tools/README.md Resolved

Comment thread .agents/skills/implementing-mcp-tools/SKILL.md Resolved

webjunkie force-pushed the chore/devex-mcp-tool-docs-cleanup branch from fc8b4c9 to bda6af8 Compare October 5, 2026 10:48

posthog Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread services/mcp/src/tools/README.md Resolved

webjunkie added 3 commits October 5, 2026 12:59

chore(mcp): point new features at the MCP feature list

32c5974

The hand-written tools guide sent authors to the repository README, but the MCP feature list lives in services/mcp/README.md.

webjunkie force-pushed the chore/devex-mcp-tool-docs-cleanup branch from d11a99f to 32c5974 Compare October 5, 2026 11:01

webjunkie mentioned this pull request Oct 5, 2026

feat(mcp): derive scopes and annotations, add tools.yaml hooks #111811

Merged

posthog Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread services/mcp/src/tools/README.md Resolved

coderabbitai Bot reviewed Oct 5, 2026

View reviewed changes

webjunkie added the stamphog Request AI approval (no full review) label Oct 5, 2026

stamphog Bot approved these changes Oct 5, 2026

View reviewed changes

trunk-io Bot merged commit 05196bd into master Oct 5, 2026

trunk-io Bot deleted the chore/devex-mcp-tool-docs-cleanup branch October 5, 2026 14:50

Conversation

webjunkie commented Oct 5, 2026

Problem

Changes

How did you test this code?

Release status

Docs update

🤖 Agent context

Uh oh!

posthog Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦔 PostHog Review reviewed this pull request

Uh oh!

github-actions Bot commented Oct 5, 2026

🤖 CI report

Uh oh!

posthog Bot commented Oct 5, 2026

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

PostHog Review

Uh oh!

Uh oh!

webjunkie commented Oct 5, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Architecture Summary

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

PostHog Review

Uh oh!

Uh oh!

trunk-io Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

PostHog Review

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

stamphog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

trunk-io Bot commented Oct 5, 2026

Uh oh!

deployment-status-posthog Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

posthog Bot commented Oct 5, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 5, 2026 •

edited

Loading

coderabbitai Bot commented Oct 5, 2026 •

edited

Loading

trunk-io Bot commented Oct 5, 2026 •

edited

Loading

deployment-status-posthog Bot commented Oct 5, 2026 •

edited

Loading